Efficient Self-attention with Relative Position Encoding for Electric Power Load Forecasting

Abstract To effectively mine historical data information and improve the accuracy of short-term load prediction, this paper aims at characteristics time series nonlinear power load. Deep learning for forecasting has received a lot attention in recent years, it become popular analysis electricity forecasting. Long memory (LSTM) gated recurrent unit (GRU) are specifically designed time-series data. However, due to gradient disappearing exploding problem, neural networks (RNNs) cannot capture long-term dependence. The Transformer, self-attention-based sequence model, produced impressive results variety generating tasks that demand long-range coherence. This shows self-attention could be useful modeling. In paper, efficiently model large-scale forecasting, we further design transform encoder with relative position encoding, which consists four main components: single-layer network, positional encoding module, feed-forward network. Experimental on real-world datasets demonstrate our method outperforms GRU, LSTM, original Transformer encoder.